Support images_by_name for Screen and Plate #64

will-moore · 2021-12-16T15:24:27Z

See #63.

Supports parsing Image Name to Image ID for HCS data. Image Name must be unique to a Plate (similar to Dataset).

NB: There has been a long-standing inconsistency in the way that populate metadata handles column names for PDI vv HCS.
For PDI, you start with "Image Name" column (with s type in header) in your csv. A new "Image" column is added with Image IDs.
This seems to make the most sense.
For HCS, you start with "Well" column (with well type in header), that contains the Well Names. This column is converted to a Well ID column (still named Well) and a new Well Name column is added, containing the names that were in the Well column. This is confusing!

In this case, so as not to have different handling of Image Name columns for HCS vv PDI, we need an Image Name column (type: s) alongside a Well column (type: well), even though they both contain names!

E.g. screen.csv

# header plate,well,s,s
Plate,Well,Image Name,Drug_name
plate1,A1,well 1 field 1, DMSO
plate1,A1,well 1 field 2, DMSO
plate1,A2,well 2 field 1, Drug1
plate1,A2,well 2 field 2, Drug1

E.g. plate.csv

# header well,s,s
Well,Image Name,Drug_name
A1,well 1 field 1, DMSO
A1,well 1 field 2, DMSO
A2,well 2 field 1, Drug1
A2,well 2 field 2, Drug1

TODO: add tests...

snoopycrimecop · 2021-12-22T00:52:40Z

Conflicting PR. Removed from build OMERO-plugins-push#988. See the console output for more details.
Possible conflicts:

PR Support ROI table on Dataset, for table with ROI ID column #62 will-moore 'Support ROI table on Dataset, for table with ROI ID column'
- src/omero_metadata/populate.py

--conflicts

will-moore · 2022-01-07T16:56:16Z

I think I need to get #62 merged ahead of this PR... Will look at that now...

snoopycrimecop · 2022-01-08T14:01:09Z

Conflicting PR. Removed from build OMERO-plugins-push#1005. See the console output for more details.
Possible conflicts:

PR Support ROI table on Dataset, for table with ROI ID column #62 will-moore 'Support ROI table on Dataset, for table with ROI ID column'
- src/omero_metadata/populate.py

~~--conflicts~~ Conflict resolved in build OMERO-plugins-push#1383. See the console output for more details.

will-moore · 2022-06-13T08:42:43Z

Closing for now to avoid more conflicts...

JensWendt · 2023-01-03T11:19:29Z

Will this be re-opened at some point?
Still hoping for the possibility to add ROI specific measurements to multiple images (aka positions) in one well.

will-moore · 2023-01-04T12:43:15Z

Re-opened and fixed merge conflicts. Hopefully tests will pass (not run locally yet).

@JensWendt it would be a great help if you can test this (see description) and let us know if it's working for you?
Thanks!

JensWendt · 2023-01-04T12:48:48Z

Heyo @will-moore,
thank you!!
Will test it 👍
Which description? The one in the first post?

will-moore · 2023-01-04T13:26:30Z

Yes, with the screen.csv and plate.csv examples. Thx

JensWendt · 2023-01-06T08:39:30Z

Heyo @will-moore ,

I did some testing.
With the CLI omero metadata populate ... and as well with the server side script Populate Metadata.
I simply exchanged the current populate.py in /opt/omero/venv_server/lib/python3.6/site-packages/omero_metadata with your changed one from this branch. I think this was the correct way to test it!?
Everything works as expected with one exception.
screen.csv will produce the expected omero-table but will throw the warning WARNING:omero_metadata.populate:PlateColumn is unimplemented.
The result looks like this, without the Plate column hyperlink functionality.

From my perspective this is not a big problem as long as the table is correct.
Good job!
Thank you!!

sbesson

A few questions on the implementation but no concern from my side.

Soince the tests have been modified, have we also have validated that the current workflow i.e. being able to annotate plate/screen without any image column is preserved? And is it still tested?

sbesson · 2023-01-09T13:58:39Z

src/omero_metadata/populate.py

@@ -742,18 +750,22 @@ def _load(self):
        self.images_by_id = dict()
        images_by_id = dict()

+        self.images_by_name = dict()
+        images_by_name = dict()


What's the goal of this line?

That creates a dictionary that is assigned to self.images_by_name[self.target_object.id.val] and is then passed to the method below where it's populated. This is just following the same pattern as for images_by_id = dict() in the lines above.

src/omero_metadata/populate.py

JensWendt · 2023-01-18T13:10:55Z

I was playing around with ROI specific metadata for whole projects/screens.
It works but - again - gives me the warning WARNING:omero_metadata.populate:PlateColumn is unimplemented. for a .csv with a Plate column containing plate names.
The bulk annotations looks like this:

No hyperlink functionality for either Plate or ROI. Again, no problem for me, but I thought it might be interesting to report.

If the pull request goes through maybe also adapt the readme.rst line
"~~If the target is an Image or a Dataset,~~ a CSV with ROI-level..."
Because now it seems to work on every level 👍

abhamacher · 2023-02-06T14:07:51Z

Hi everyone,

I also did some testing this morning with the following combinations of source csv data containing either:

plate, well
plate, well, image
plate, well, image, roi (only with the current github version of populate.py)

Except of the warning message, that Jens already mentioned, I would like to point out the following concerns:

When populating image-based data an "Image Number" is added to the OMERO.table. But this number is not the Image ID, which I find a bit confusing.
When populating image-based data, OMERO.parade can properly load individual image data for example in the filter menu (see Example 68...366 per image) but the dots in the parade plot do not cover all images, but rather wells only, which is inconsistent.

(as a side note, with the same OMERO.table plots in parade-crossfilter show all individual image dots)

Based on concern no. 3, using parade one cannot see if the data shown is coming from wells or from images. Typically someone would populate the OMERO.table but another or multiple persons would browse through the data, which are not aware of the underlying data structure. So this is something that should be visible somehow to the user.
When populating roi-based data, the activity log returns an irritating message ("Failed to get results"), although the OMERO.table seems to be generated properly.
Issue no. 2-no. 4 also apply to roi-based data.
I'm not a big fan of the order of columns. For example, placing well id at the beginning of the OMERO.table but well name and plate name at the very end? I typically use Cellprofiler output for uploading metadata, which has a lot of columns and therefore results into lots of scrolling to the right, if I want a readable well name, for example.
Another side note for @will-moore and parade-crossfilter: when selecting one row of the table view panel of roi-based data, only one specific image is shown in the image preview panel (which is therefore fine). However, image-based data results in the preview of all fields of the wells, instead of a single image (which should be basically selected in the table view by selecting a single row).

I hope my notes are not too confusing and a bit helpful.

BR, Anna

The script is working fine, so it looks like these are invalid warnings but it is hard to understand the working of the script

will-moore · 2023-02-08T16:00:20Z

@abhamacher "Image Number" - do you mean the "Image" column? That is definitely supposed to be the Image ID. If it wasn't then the script is failing badly. Do you have some sample data (csv)?

I think omero-parade doesn't handle multiple Images per Well and certainly not multiple ROIs per Image.

omero-metadata always adds Columns at the end rather than inserting them.

I don't quite understand what 8) is about. Could you create a new issue for this on that repo (and same for any others above if you want, on their respective repos).

In the last commit above, I have reduced the logging level for warnings.

abhamacher · 2023-02-10T12:53:51Z

Hi Will,

I apologize for mixing up topics and bringing some confusion to this thread. As you can see, I added parts of the mentioned issues at the parade and parade-crossfilter repo. About the "ImageNumber" -- that was a mean one, it actually came from the original Cellprofiler data. Sorry, about that, so it has nothing to do with omero-metadata.

omero-metadata always adds Columns at the end rather than inserting them.

Well, still this is not very handy/ well readable for users. Especially because I add for example the plate and well names to the upload csv-file as the very first columns, but omero-metadata exchanges them to IDs and adds the names to the end of my table. Why can't the names just remain where they are (at the beginning) and the IDs being added to the end of the table?

Thanks, Anna

will-moore · 2023-02-10T13:07:15Z

Hi Anna,
yes I agree that's annoying behaviour for HCS data (as I described in this PR description) - The behaviour for Project/Dataset/Image is better - in that case the "Image Name" column is unchanged and an "Image" ID column is simply added.
I have previously created an issue for this at #29
Feel free to add comment there.

JensWendt · 2023-02-14T13:43:01Z

Hi @abhamacher ,

regarding point 5) --> I've been running into this issue also with other scripts.
See my recent thread https://forum.image.sc/t/failed-to-get-results-error-while-running-scripts/76459 and the two other threads referenced in the first post there.
It seems to be an issue with processes which take a lot of time or involve a lot of "entities", that at some point something goes wrong and you may or may not get the desired end result after seeing the "Failed to get results" statement.

snoopycrimecop · 2024-01-16T00:56:43Z

Conflicting PR. Removed from build OMERO-plugins-push#372. See the console output for more details.
Possible conflicts:

PR Prevent users from creating columns with python keyword names #55 kkoz 'Prevent users from creating columns with python keyword names'
- src/omero_metadata/populate.py
Upstream changes
- src/omero_metadata/populate.py

--conflicts

will-moore added 4 commits December 16, 2021 14:54

Support images_by_name for Screen and Plate

464680f

Handle images_by_name when target_object is Plate

1bb59af

add testing of Image Name to ID in Plate

33a9554

flake8 fixes

803f97c

will-moore mentioned this pull request Jan 7, 2022

Importing plate data per image #63

Open

sbesson mentioned this pull request Jan 13, 2022

Add Image IDs workflow to README #66

Closed

Merge remote-tracking branch 'origin/master' into plate_images_by_name

4e5e8b9

snoopycrimecop mentioned this pull request Jun 6, 2022

Fix header detection for tables with sparse numerical data #77

Merged

will-moore closed this Jun 13, 2022

will-moore reopened this Jan 4, 2023

ome deleted a comment from snoopycrimecop Jan 4, 2023

Merge remote-tracking branch 'origin/master' into plate_images_by_name

90a66d3

will-moore force-pushed the plate_images_by_name branch from fa13d53 to 90a66d3 Compare January 4, 2023 12:25

JensWendt mentioned this pull request Jan 6, 2023

tickbox “Fields:” in the plate layout results in a white screen will-moore/parade-crossfilter#16

Open

sbesson reviewed Jan 9, 2023

View reviewed changes

will-moore added 2 commits February 8, 2023 14:50

Add comments to try and help understanding

edc0853

Use log.debug() instead of warn() or info()

78e5553

The script is working fine, so it looks like these are invalid warnings but it is hard to understand the working of the script

This was referenced Feb 10, 2023

Multiple fields per well ome/omero-parade#61

Open

Selecting a row in table panel loads more than the specific image for preview in case of image-/roi-based OMERO.table input will-moore/parade-crossfilter#38

Open

snoopycrimecop mentioned this pull request Jan 13, 2024

Fix flake8 issue raised following the Rocky Linux 9 image upgrade #84

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support images_by_name for Screen and Plate #64

Support images_by_name for Screen and Plate #64

will-moore commented Dec 16, 2021 •

edited

Loading

snoopycrimecop commented Dec 22, 2021

will-moore commented Jan 7, 2022 •

edited

Loading

snoopycrimecop commented Jan 8, 2022 •

edited

Loading

will-moore commented Jun 13, 2022

JensWendt commented Jan 3, 2023

will-moore commented Jan 4, 2023

JensWendt commented Jan 4, 2023

will-moore commented Jan 4, 2023

JensWendt commented Jan 6, 2023

sbesson left a comment

sbesson Jan 9, 2023

will-moore Jan 10, 2023

JensWendt commented Jan 18, 2023

abhamacher commented Feb 6, 2023

will-moore commented Feb 8, 2023

abhamacher commented Feb 10, 2023

will-moore commented Feb 10, 2023

JensWendt commented Feb 14, 2023

snoopycrimecop commented Jan 16, 2024

Support images_by_name for Screen and Plate #64

Are you sure you want to change the base?

Support images_by_name for Screen and Plate #64

Conversation

will-moore commented Dec 16, 2021 • edited Loading

snoopycrimecop commented Dec 22, 2021

will-moore commented Jan 7, 2022 • edited Loading

snoopycrimecop commented Jan 8, 2022 • edited Loading

will-moore commented Jun 13, 2022

JensWendt commented Jan 3, 2023

will-moore commented Jan 4, 2023

JensWendt commented Jan 4, 2023

will-moore commented Jan 4, 2023

JensWendt commented Jan 6, 2023

sbesson left a comment

Choose a reason for hiding this comment

sbesson Jan 9, 2023

Choose a reason for hiding this comment

will-moore Jan 10, 2023

Choose a reason for hiding this comment

JensWendt commented Jan 18, 2023

abhamacher commented Feb 6, 2023

will-moore commented Feb 8, 2023

abhamacher commented Feb 10, 2023

will-moore commented Feb 10, 2023

JensWendt commented Feb 14, 2023

snoopycrimecop commented Jan 16, 2024

will-moore commented Dec 16, 2021 •

edited

Loading

will-moore commented Jan 7, 2022 •

edited

Loading

snoopycrimecop commented Jan 8, 2022 •

edited

Loading